
Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 





@ Publication number : 0 462 905 A2 



EUROPEAN PATENT APPLICATION 



@ Application number: 91401677.9 
(22) Date of filing : 20.06.91 



© int. CI. 5 : H04N 1/21 



(30) Priority: 20.06.90 JP 161870/90 

(S) Date of publication of application : 
27.12.91 Bulletin 91/52 

(3) Designated Contracting States : 
DE FR GB IT NL 

(71) Applicant : SONY CORPORATION 

7-35, Kitashinagawa 6-chome Shinagawa-ku 
Tokyo (JP) 



(72) Inventor : Kojima, Kiyonobu, c/o SONY 
CORPORATION 

7-35, Kitashinagawa 6-chome, Shinagawa-ku 
Tokyo (JP) 

(m) Representative : Thevenet, Jean-Bruno et at 
Cabinet BEAU DE LOMENIE 55 rue 
d 'Amsterdam 
F-75008 Paris (FR) 



(g) Electronic camera and image reproducing device therefor. 

(57) An electronic camera includes a sensor for detecting an image of a subject, a sensor (2) for detecting 
a three dimensional position and azimuth of the camera, a focusing amount detector (6) and a zoom 
amount detector (7), so that the image is recorded on a record medium (4a) together with information 
concerning the three dimensional position and azimuth, the focusing amount and the zoom amount An 
image reproducing device for the camera reproduces the image recorded on the record medium (4a) on 
the basis of the information so that an interrelationship between a plurality of images recorded on the 
record medium (4a) is clearly reproduced. Accordingly, the camera and the reproducing device are 
suitable for combining the recorded images so as to prepare a panoramic image and to locate a desired 
article within the image which may be accommodated together with different articles. 
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BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

This invention relates to an electronic camera 
which is usable for storing image data on a record 
medium together with auxiliary data including 
information concerning a three dimensional 
position/direction of the camera, a focusing and a 
zoom amount of the camera. The invention further 
relates to an image reproducing device for 
reproducing the image data stored on the record 
medium in association with the auxiliary data such 
that an interrelationship between a plurality of images 
being recorded is clarified. 

2. Description of the Background Art 

In general, many kinds of cameras have been 
used as an image recording device for storing a 
reproduceable image of a subject on a record 
medium. Recently, there has been proposed an 
electronic camera usable for electronically recording 
the image on a record medium such as a floppy disk, 
and/or a memory card. In the electronic camera or a 
video camera, image information on the record 
medium is recorded electronically and is therefore 
conveniently applicable to various fields and media 
formats. 

However, despite high reproduceability, 
conventional cameras do not have any mechanism for 
recording information which relates to 
interrelationship between a plurality of images stored 
on the record medium except for the recording date 
and time. Therefore, when the same portion of the 
same subject appears in separate images stored on 
the record medium, the camera does not have any 
means which to register like portions of respective 
images of the same subject nor represent combined 
images smoothly and clearly. 

FIG. 6 illustrates a method for preparing a 
panoramic view using a conventional camera. The 
camera 101, held in place, records separate images 
102a, 102b, 102c and 102d continuously, while 
changing the camera angle, such that respective 
images are partially overlapped with adjacent ones. 

The images 102a, 102b, 102c, 102d must be 
combined to produce the panoramic view, by carefully 
overlapping the same portions of the same subjects 
which appears in the respective adjacent images. 

However, in the conventional method, difficulty is 
encountered in respect to the following, multiple 
image recording should be done: 

(1) at the same magnification, 

(2) in series without any missing image portions, 

(3) from the same position without any change in 
camera direction. 

Generally, in the case of continous shots at a given 



place, it is required to record an image in detail at a 
high magnification or from various different angles 
relative to the subject. In conventional systems the 
above-mentioned points are not completely satisfied. 

5 For example, for photographing acheological 
excavations, or the like, continuous panoramic 
images are desirable at various scales and from 
various viewpoints. 

In order to eliminate the above inconvenience. 

10 conventional cameras prepare a memorandum upon 
shooting, in which Information including a position of 
the camera, a direction thereof and the like are noted. 
When such information cannot be exactly noted, a 
rough schematic drawing imcompletely illustrating the 

15 position and direction is prepared. 

As described on the above, conventionally 
proposed cameras are not provided with a 
mechanism for detecting and recording information 
about a three dimensional position and direction of the 

20 camera. Therefore, the camera may not be utilized, 
for example, for searching articles stored in an office 
or warehouse. 
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SUMMARY OF THE INVENTION 



It is an object 6f the present invention to eliminate 
the disadvantages mentioned above and to provide 
an electronic camera which records an image on a 
record medium together with auxiliary information 

30 concerning a three dimensional position of the 
camera, an azimuth thereof, and the like. It is another 
object of the invention to provide an image 
reproducing device which reproduces the image 
stored on the record medium in association with such 

35 auxiliary information so that an interrelationship 
between a plurality of recorded images is smoothly 
integrated. 

In order to accomplish these objects, there is 
provided an electronic camera having a lens section 

40 capable of focusing incident light from a subject for 
recording, which includes first detecting means for 
detecting an image in response to incident light 
passing through the lens section, on the basis of 
which image data is formed, and second detecting 

45 means for detecting a three dimensional position in 
which the camera is located and an azimuth of the 
camera located therein. Third detecting means for 
detecting a focusing amount from which a distance 
between the camera and the subject is derived, is 

so mounted on the lens section. Fourth detecting means 
for detecting a zoom amount from which a zooming 
magnitication is derived, is mounted on the lens 
section. Further, the camera includes a processing 
unit for generating an auxiliary data corresponding to 

55 the image data, on the basis of the three dimensional 
position and azimuth detected by the second 
detecting means, the focusing amount and the zoom 
amount detected by the third detecting means and the 
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fourth detecting means. The processing unit also 
converts the auxiliary data into a magnetically 
recordable one. Furthermore, magnetic recording 
means are provided for recording the image data and 
the auxiliary data on a record medium. The magnetic 
recording means is electrically connected to the first 
detecting means and through the processing unit to 
the second, third, fourth detecting means. 

In addition, there is provided an image 
reproducing device for an electronic camera 
according to the invention, which includes a disk drive 
for reading out the image data and auxiliary data 
recorded on the record medium, and a computer for 
editing, processing or searching digital signals of the 
image data read out by said disk drive, on the basis 
of digital signals of the auxiliary data read out by the 
disk drive. The reproducing device further includes a 
memory in which digital signals of the read-out image 
data and the read-out auxiliary data are stored, and a 
frame memory in which the digital signals edited, 
processed or searched by the computer are stored. 
Further, the reproducing device is provided with a 
display unit for displaying images according to the 
image data, and a mouse through which user 
commands for editing, processing or searching the 
digital signals of the read-out image data are 
transmitted to the computer. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention will be understood from the 
detailed description given hereinbelow and from the 
accompanying drawings of the preferred embodiment 
which are given for explanation and understanding 
only and are not intended to imply limitation of the 
invention. 

In the drawings: 

FIG. 1 is a perspective view of one embodiment 
of an electronic camera according to the 
invention; 

FIG. 2 is a block diagram of the camera shown in 
FIG. 1; 

FIG. 3 is a block diagram of one embodiment of 
an image reproducing device according to the 
invention; 

FIGS. 4 and 5 are diagrams explanating a search 
operation of the device according to the invention; 
and 

FIG. 6 is a diagram for explanating an image- 
combining operation of an electronic camera 
according to the prior art. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

The preferred embodiment of an electronic 
camera according to the present invention will now be 
described with reference to the accompanying 



drawings. 

The embodiment is indicated as an electronic 
camera utilized as a still picture recording device, 
though it will be noted that the present invention may 
5 be embodied in various other ways. 

Referring to FIG. 1 , an electronic camera 1 has a 
camera housing 1a, a lens section 1b having a lens 
and a shutter button 1c. The housing 1a includes a 
three dimensional position- and azimuth-sensor 2, an 
10 image sensor 3, a magnetic recording unit 4, and a 
processing units. The lens section 1b including a lens 
controlling unit is provided with a focusing amount 
detector 6 and a zoom amount detector 7. As shown 
in FIG. 2, the shutter button 1c and the image sensor 
15 3 are electrically connected to the magnetic recording 
unit 4. The image sensor 3 detects an image focussed 
by manual- or auto-focus operation, through the lens 
section 1 b in response to depression of the shutter 
button 1c and transmits an image signal to the 
20 magnetic recording unit 4 where image data 
corresponding to the image signal is magnetically 
recorded on a record medium 4a such as a floppy 
disk. Further, each of the three dimensional position- 
and azimuth-sensor 2, the focusing amount detector 
25 6 and the zoom amount detector 7 is electrically 
connected through the processing unit 5 to the 
magnetic recording unit 4. The three dimensional po- 
sition- and azimuth-sensor 2 detects a three 
dimensional position, in which the camera 1 is 
30 located, and an azimuth of the camera 1. Signals 
corresponding to the detected position and azimuth 
are transmitted to the processing unit 5; The focusing 
amount detector 6 detects a focusing amount and 
transmits a signal indicative thereof to the processing 
35 unit 5 where a distance between the camera and the 
subject is calculated on the basis of the detected 
focusing amout. The zoom amount detector 7 detects 
a zoom amount and transmits a zoom amount signal 
to the processing unit 5 which calculates zoom 
40 magnification or the basis of the detected zoom 
amount. When signals corresponding to the detected 
focusing amount and the detected zoom amount are 
transmitted to the processing unit 5, the processing 
unit 5 generates an auxiliary information signal which 
45 is associated with the image now detected at the 
image sensor. The auxiliary information signal is 
transmitted to the magnetic recording unit 4 and then 
stored as auxiliary data together with the associated 
image data on the record medium 4a. Sensors usable 
50 as the sensor 2 of the present invention include a 
global positioning system which detects three 
dimensional position and azimuth by analyzing radio 
waves received from a communications satellite or 
the like, and a geomagneticsensorwhichdetects only 
55 azimuth. Since the global positioning system has an 
error range of 10 - 20 m upon detecting a three 
dimensional position, it is desirable to employ an 
ultrasonic sensor when higher sensing accuracy is 
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required. In this case, the ultrasonic sensor detects 
respective ultrasonic waves emitted from at least 
three ultrasonic sources arranged at predetermined 
positions and analyzes the three dimensional position 
and the azimuth. Additionally, an electromagnetic 5 
sensor is applicable, which has three coiis wound in 
separate three directions. The electromagnetic 
sensor detects a change in an amount of a current 
flowing through each coii when receiving radio waves 
emitted from a source located at a predetermined 10 
position. In other words, in the electromagnetic 
sensor, the three dimensional position and the 
azimuth are detected as change in the amount of 
current amount in respective coils thereof. 

One preferred embodiment of an image 15 
reproducing device for the aforementioned electronic 
camera, according to the present invention will be 
described hereinbelow. FIG. 3 is a block diagram of 
the image reproducing device. The image 
reproducing device 10 includes a disk drive 11 for 20 
reading out the image data and the associated 
auxiliary data which are recorded on the record 
medium 4a. The image data and the auxiliary data 
read out from the record medium 4a by the disk drive 
1 1 are transmitted to the computer 1 4. In the case that 25 
the read-out image data is in the form of analog 
signals, the disk drive 1 1 is connected to the computer 
14 via an A/D convertor 12 which converts analog 
signals to digital signals. Digital signals of the read- 
out data or digital signals digitized by the A/D 30 
convenor 12 transmitted to the computer 14 are 
forwarded to a memory 13 in which the digital signals 
are temporarily stored. Further, the computer 14 is 
connected to a frame memory 15 and a mouse 16 
through which user commands are transmitted to the 35 
computer 14. Depending upon the user command, the 
computer 14 derives digital signals stored in the 
memory 13 and edits, processes or searches the 
image data on the basis of the associated auxiliary 
data stored in the memory 1 3. The image data edited, 40 
processed or searched by the computer 14 is 
transmitted to the frame memory 15 and stored 
therein. The frame memory 15 is connected to a 
display 17. After the digital signals of the image data 
are converted into analog signals, the image is 45 
reproduced on the display 17. 

The auxiliary data stored in the record medium 4a 
together with the image data includes information 
such as the three dimensional position at which the 
camera 1 is located, the azimuth of the camera 1 , the 50 
distance between the camera 1 and the subject and 
an angle of visibility. These are calculated by either 
the processing unit 5 of the camera 1 or the computer 
14 of the reproducing device 10 on the basis of 
respective values detected by the three dimensional 55 
position- and azimuth-sensor 2, the focusing amount 
detector 6 and the zoom amount detector 7. The 
computer 14 performs various processing of the 



image data on the basis of the auxiliary data. For 
instance, the computer 14 combines a .plurality of 
separate images in order to produce a panoramic 
image, or reduces the respective image data to totally 
indicate the images in a chart form on the display. The 
computer 14 has a capability to zoom in or out on a 
portion of the image to be show same in enlarged or 
reduced scale, and to mark the subject image area to 
be affected. In addition, the computer 14 may search 
for an article, conforming to specific image data in a 
warehouse or an office, for example. 

OPERATION 

The aforementioned embodiment of the present 
invention will be described hereinbelow with respect 
to its operation and function. 

The image sensor 3 of the electronic camera 1 
detects an image in response to incident light passing 
through the lens section 1b. The detected image is 
transmitted to the magnetic recording unit 4 where the 
image is recorded as image data on the record 
medium 4a. The three dimensional position- and azi- 
muth-sensor 2 detects a three dimensional position 
and an azimuth of the camera 1 which is located in a 
predetermined position. The focusing amount 
detector 6 and the zoom amount detector 7 are 
provided in the camera 1 and detect a focusing 
amount and a zoom amount of the camera 1, 
respectively. The three dimensional position, the 
azimuth, the focusing amount and the zoom amount 
detected by the respective sensors and the detectors 
are transmitted to the processing unit 5. The 
processing unit 5 calculates a coordinate of the 
position of the camera 1 , the azimuth of the camera, 
a distance between the camera 1 and the subject, or 
focal point and zoom magnification value on the basis 
of the data transferred from the sensors 2. 6, 7. The 
processing unit 5 further converts the calculated 
values into magnetically recordable signals and 
transmits the signals to the magnetic recording unit 4. 
The transmitted signals are recorded on the record 
medium 4a as auxiliary data together with the image 
data. 

Subsequently, the image data recorded on the 
record medium 4a is reproduced by the image 
reproducing device 10. In this embodiment, a variety 
of reproductions of the image data are performed by 
associating with the auxiliary data, in addition to usual 
reproduction of the image. The image data and the 
auxiliary data which are recorded in association on 
the record medium 4a are transmitted to the computer 
14 so that the image data may rearranged or 
converted by the computer 14 into three dimensional 
image data. 

For example, in the case of preparation of a 
panoramic image, a plurality of images with various 
magnification are converted to those having the same 
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magnification, based on the zoom magnification 
contained in the auxiliary data for each image. The 
converted image data is stored in the frame memory 
15 in association with the position coordinates and the 
azimuth of the auxiliary data. The image data stored 
in the frame memory 15 is read out and indicated on 
the display 17 in a scrollable manner to form a 
panoramic image. In this case, if desired, the image 
may be zoomed so as to be magnified. Further, if the 
image data stored in the frame memory 15 has a large 
zoom magnification, detail of the image subject can 
be conveniently observed. 

Diagrams (a), (b), (c) of FIG. 4 are directed to a 
process of searching for an article stored in such a 
place as a warehouse. Referring to the diagram (a) of 
FIG. 4, reduced images of articles are represented 
together in the form of a table on the display 1 7. A user 
finds a desired article in the table and marks the article 
by moving a cursor with the mouse 16. The computer 
14 calculates a coordinate of the position of the 
marked article in accordance with the auxiliary data 
associated with the image data, which includes a 
three dimensional position of the camera, an azimuth 
of the camera and a distance between the camera and 
the article. A point based on the coordinate of the 
position is plotted in a whole image in which the article 
in concern appears together with other surrounding 
articles, by placing a mark such as a star as illustrated 
in the diagram (b) of FIG. 4. To this end, the desired 
article is correctly searched from other ones 
throughout a total recorded image. As shown in the 
diagram (c) of FIG. 4, a star symbol is positioned on 
the desired article which is present, in this example, 
together with other articles in a carton or box. As 
shown in each of the diagrams (b) and (c), the article 
in concern is in a position marked with the star symbol. 
Thus, the desired article is searched out from a 
plurality of articles which are accommodated in the 
same place. 

In the search for the desired article as mentioned 
above, diagrams (a), (b), (c) of FIG. 5 illustrate a 
process for judging whether or not the target article 
exists in an image in which a plurality of articles 
appear. The diagram (a) is a top view showing an 
arrangement of the camera 1 and an article 20 which 
are on the same plane. The diagram (b) is a side view 
showing the same arrangement as that of the diagram 
(a). The diagram (c) shows the coordinate of the 
position of the article plotted in an image which is 
formed by pixel w in the transverse direction and pixel 
h in the longitudinal direction. In the diagrams, an X- 
axis extends perpendicular to the Y-axis in the 
horizontal direction so as to define a fiat plane. A Z- 
axis extends from a point of intersection of the X-axis 
and the Y-axis in the vertical direction and 
perpendicular to the flat plane. If the point of 
intersection is taken as an origin, the coordinate of the 
position of the article is represented as (xq, y 0 , Zo) and 
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the coordinate of the position of the camera is 
represented as (x, y, z) in the image. A direction 
vector of the camera is represented as (a, b, c). A 
horizontal direction angle of view of the camera is 
represented as a while a vertical direction angle is 
represented as p. In this case, it is assumed that the 
coordinate of the position of the article exists in the 
image when the following expression is satisfied: 
a(Xp-x) + b(y 0 -y) > . , a, 
Va2 + b2.V(Xo-x)2 + (y 0 -y)2 V 



Va^ + c2-V(x G -x)2 + (Zo-z) 2 V 



(1) 
(2) 



where, 0 Q <a, p<180°. 

Further, as shown in the diagram (c) of FIG. 5, 
when the center of an image is taken as an origin 
point, the position of the article is plotted by the 
coordinate (w' f h') in the image which is obtained by 
the following expressions: 

w ~ 2a cosr ViFTb^CXo-x)* +(y 0 -y) 2 

h' - JL - a(*o- x ) + cfer*) 

n 2p >/a2 + c2^(x 0 -x)2+.(z 0 -z)2 

With respectto w' of the above expression, if a(y 0 - 
y)-b(x 0 -x) has a positive value, cos- 1 .has a positive 
value. Conversely, if a(y 0 -y)-b(xo-x) has a negative, 
cos -1 has a negative value. 

With respect to h' of the expression, if a(zo-z)- 
c(x 0 -x) has a positive value, cos- 1 has a positive value. 
If a(Zo-z)-c(x 0 -x) has a negative value, cos- 1 has a 
negative value. 

The present invention is not limited only to 
electronic camera for a still photograph recording but 
is further applicable for a moving-image recording 
device such as a video camera. In addition, according 
to the present invention, the three dimensional 
position is accurately detected by an image 
processing or a stereopsia generation which is 
realized by collating and correcting the data 
concerning a position of the camera and the subject. 
To this end, simulation of a free navigation in three 
dimensional space is realized in addition to the search 
for the article as mentioned on the above. Although 
the preferred embodiment of the invention utilizes all 
information of the auxiliary data concerning the 
position and direction of the camera, the focusing 
amount and the zoom amount, it will be apparent that 
the invention may employ all or some of the 
information or a combination thereof. Accordingly, 
many modifications and variations can be effected by 
one skilled in the art without departing from the spirit 
or scope of the invention is defined in the appended 
claims. 

As is obvious from the aforementioned 
description of the preferred embodiment of the 
invention, the electronic camera is capable of 
immediately and simultaneously recording a plurality 
of images which are concerned with each other with 



9 EP 0 462 905 A2 10 



respect to the same subject. Further, the image 
reproducing device reproduces the image on the 
basis of the auxiliary data. Accordingly, depending 
upon such a data storage function of the camera, a 
position of each article accommodated in such a place 
as a warehouse can be surely found out without any 
direct visual detection of the article so that convenient 
arrangement or storage of articles is effectively 
realized. Moreover, the individual images being 
recorded are three-dimensionally reproduceable and 
usable for a wide variety of applications. 



Claims 

1. An electronic camera, characterized in that it 
comprises; 

a lens section (1b) capable of focusing 
incident light having an image of a subject to be 
recorded; 

first detecting means (3) for detecting the 
image of the subject in response to the incident 
light passing through the lens section (1b), on the 
basis of which image data is formed; 

second detecting means (2) for detecting a 
three dimensional position at which the camera 
(1) is located and an azimuth of the camera; 

third detecting means (6) for detecting a 
focusing amount from which a distance between 
the camera (1) and the subject is derived, said 
means being mounted on the lens section (1b); 

fourth detecting means (7) for detecting a 
zoom amount from which zoom magnification is 
derived, said means being mounted on the lens 
section (1 b); 

a processing unit (5) for generating 
auxiliary data in association with the image data, 
on the basis of the three dimensional position and 
the azimuth detected by said second detecting 
means (2), the focusing amount and the zoom 
amount detected respectively by said third and 
fourth detecting means (6 J); and 

magnetic recording means (4) for 
recording the image data and the auxiliary data 
on a record medium (4a); said magnetic recording 
means (4) being electrically connected to said 
first detecting means (3) and through said 
processing unit (5) to said second, third and 
fourth detecting means (2,6,7). 

2. The electronic camera according to claim 1, 
wherein said lens section (1b) includes control 
means for controlling the focusing amount and 
the zoom amount, 

3. An image reproducing device for the electronic 
camera (1) according to the claim 1, comprising: 

a disk drive (11) for reading out the image 



data and the auxiliary data which are recorded on 
the record medium (4a); 

a computer (14) for editing, processing or 
searching the image data read out by said disk 
drive (1 1 ), on the basis of the auxiliary data read 
out by said disk drive (1 1); 

a memory (13) in which the read-out image 
data and the read-out auxiliary data are stored; 

a frame memory (15) in which the image 
data edited, processed or searched by said 
computer (14) are temporarily stored; and 

a display unit (17) for displaying the image 
data stored in said frame memory (15). 



15 4. The image reproducing device according to claim 
4, wherein said computer (14) is used as the 
processing unit (5) for the electronic camera (1). 
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